Chinese Word Segmentation in ICT-NLP

Chinese word segmentation is always much accounted of in ICT-NLP. In this bakeoff, two different systems in ICTNLP participated. The one is SYSTEM_#1 evaluated in three tracks -PK-close, MSR-close and MSR-open, and SYSTEM_#2 PK-open. Through this bakeoff , the development of Chinese segmentation is learned and the problems are found in our systems. 1 System Description Two different systems in ICT-NLP participated the second bakeoff. 1.1 SYSTEM_#1 The SYSTEM_#1 is implemented mainly based on the log-linear model CRFs(Conditional Random Fields). CRFs are arbitrary undirected graphical models trained to maximize the conditional probability of the desired outputs given the corresponding inputs. We cast the segmentation as one of sequence tagging. The conditional probability for the tag sequence given a input Chinese sentence is defined by a linear-chain CRF with parameters 1 2 ... n T t t t 1 2 ... n C c c c 1 2 ... m to be 1 1 1 ( | ) exp ( , , , ) n m m i i i m c P T C f t t C i Z Where c Z is the per-input normalization that makes the probability of all state sequences sum to one. is a feature function which is can be any real number. 1 ( , , , ) m i i f t t C i The most probable tag sequence for an input C, * arg max ( | ) T T P T C is determined using the Viterbi algorithm, An Nbest list of tagging sequences is obtained using modi-fied Viterbi algorithm. Six tags according to the different positions of one character in a word are used in this model, such as #START(beginning of one sentence), B(beginning of one word), M(middle of one word), E(end of one word), and #END(end of one sentence). The feature templates used in this model are listed in Table 1. Description Feature current state , i i t c current & previous states 1, , i i t t ci current & two previous states 2 1 , , , i i i t t t ci state transitions 1, i i t t second previous character 2 , i i t c previous character 1 , i i t c next character 1 , i i t c second next character 2 , i i t c previous two characters 2 1 , , i i i t c c next two characters 1 2 , , i i i t c c previous current & next character 1 1 , , , i i i i t c c t current and previous character 1 , , i i i t c c current and next character 1 , , i i i t c c character types of current & two previous & two next characters 2 1